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This  booklet  is  part  of  the  Planning  and  Evaluating  Health  Information  Outreach  Projects  series, 
designed  to  supplement  Measuring  the  Difference:  Guide  to  Planning  and  Evaluating  Health 
Information  Outreach.  [1]  This  series  also  supports  evaluation  workshops  offered  through  the 
Outreach  Evaluation  Resource  Center  of  the  National  Network  of  Libraries  of  Medicine  (NN/LM). 
The  goal  of  the  series  is  to  present  step-by- step  planning  and  evaluation  methods.  Along  with 
providing  information  about  evaluation,  each  booklet  includes  a  case  study  and  worksheets  to  help 
you  with  your  outreach  planning. 

The  series  emphasizes  the  relationship  between  planning  and  evaluation  —  this  is  why  both  words 
are  part  of  the  series  title.  By  including  evaluation  in  the  planning  stage,  you  are  committing  to 
doing  it  and  you  are  more  likely  to  make  it  integral  to  the  overall  project.  Conversely,  in  planning 
the  evaluation  you  identify  outcomes,  which  in  turn  help  you  to  carefully  assess  project  activities 
and  resource  needs. 

These  booklets  are  aimed  at  librarians  —  from  the  health  sciences  sphere,  particularly  —  and 
representatives  from  community  organizations  who  are  interested  in  conducting  health  information 
outreach  projects.  We  consider  "health  information  outreach  projects"  to  be  educational 
or  awareness  activities  designed  to  enhance  community  members'  abilities  to  find  and  use 
information.  A  goal  of  these  activities  might  be  to  equip  group  members  to  better  address  their 
—  and  their  family  members'  and  peers'  —  questions  about  health.  Such  outreach  often  focuses 
on  online  health  information  resources  such  as  the  Websites  produced  by  the  National  Library  of 
Medicine.  Projects  may  also  include  other  sources  and  formats  of  health  information. 

The  first  booklet,  Getting  Started  with  Community-Based  Outreach  is  designed  for  those  who  have 
an  idea  for  working  with  their  communities  but  do  not  know  how  to  start.  It  describes  these  steps: 

1.  Find  partners  for  health  information  outreach  projects, 

2.  Learn  more  about  the  outreach  community,  and 

3.  Inventory  resources  and  assets. 

The  second  booklet,  Including  Evaluation  in  Outreach  Project  Planning,  is  intended  for  those  who 
need  guidance  in  designing  a  good  evaluation  plan.  It  discusses  how  to  do  the  following: 

1 .  Develop  an  outcomes-based  project  plan, 

2.  Develop  an  outcomes  assessment  plan, 

3.  Develop  a  pre-project  assessment  plan,  and 

4.  Develop  a  process  assessment  plan. 

The  third  booklet,  Collecting  and  Analyzing  Evaluation  Data,  will  probably  be  more 
understandable  to  those  with  some  experience  in  conducting  health  information  outreach,  but 
those  just  starting  in  health  information  outreach  also  may  find  it  useful  for  planning  their 
outreach  programs.  It  presents  these  steps  for  quantitative  methods  (processes  for  collecting  data 
and  turning  them  into  statistics)  and  qualitative  methods  (processes  for  collecting  non-numeric 
descriptive  information  and  summarizing  it): 

1 .  Design  your  data  collection  methods, 

2.  Collect  your  data, 
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3.  Summarize  and  analyze  your  data,  and 

4.  Assess  the  validity  of  your  findings. 

We  strongly  endorse  partnerships  among  organizations  from  a  variety  of  environments,  including 
health  science  libraries,  community-based  organizations,  and  public  libraries.  We  also  encourage 
broad  participation  of  members  of  target  outreach  populations  in  the  design  and  implementation  of 
the  outreach  project.  We  try  to  describe  planning  and  evaluation  methods  that  accommodate  this 
approach  to  community-based  outreach.  Still,  we  may  sound  like  we  are  talking  to  project  leaders. 
In  writing  these  booklets  we  have  made  the  assumption  that  one  person  or  a  small  group  of  people 
will  be  in  charge  of  initiating  an  outreach  project,  writing  a  clear  project  plan  and  managing  the 
evaluation  processes. 

We  also  encourage  evaluation  practices  that  adhere  to  the  Program  Evaluation  Standards 
developed  by  the  Joint  Committee  on  Standards  for  Educational  Evaluation,  which  can  be 
found  at  http://www.eval.org/EvaluationDocuments/progeval.html  [2].  The  utility  standards 
require  that  evaluation  findings  will  serve  the  information  needs  of  the  intended  users,  primarily 
those  implementing  a  project  or  those  with  some  vested  interest  in  it.  The  feasibility  standards 
direct  evaluation  to  be  cost-effective,  credible  to  the  different  groups  who  will  use  evaluation 
information,  and  minimally  disruptive  to  the  project.  The  propriety  standards  uphold  evaluation 
that  is  conducted  ethically,  legally,  and  with  regard  to  the  welfare  of  those  involved  in  or  affected 
by  the  evaluation.  Finally,  the  accuracy  standards  indicate  that  evaluation  should  provide 
technically  adequate  information  for  evaluating  a  project. 

We  sincerely  hope  that  you  find  these  booklets  useful.  We  welcome  your  comments,  which  you 
can  email  to  nnlm@u.washington.edu 
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1  Introduction 


Step  One 

Develop  an  Outcomes-Based  Project  Plan 

i 

Step  Two 

Develop  an  Outcomes  Assessment  Plan 

t 

Step  Three 

Develop  a  Pre-Project  Assessment  Plan 

i 

Step  Four 

Develop  a  Process  Assessment  Plan 

New  outreach  projects  can  be  exciting, 
especially  when  the  project  team  has 
innovative  ideas  for  achieving  outreach  or  has 
a  wave  of  enthusiasm  for  a  new  community 
partnership.  Evaluation,  believe  it  or  not,  can 
add  to  the  excitement  if  it  is  done  correctly. 

If  evaluation  is  planned  and  conducted 
systematically  throughout  the  project,  it  can 
provide  information  for  self-reflection  and 
project  improvement.  Evaluation  also  can 
help  you  provide  compelling  evidence  of 
effectiveness  of  a  well-done  project,  which 
is  of  interest  to  all  stakeholders  in  your 
project.  (A  stakeholder  is  defined  as  anyone 
who  has  some  form  of  investment  or  "stake" 
in  the  project).  You  will  be  able  to  take  full 
advantage  of  these  benefits  if  you  include 
evaluation  in  your  project  plan  from  the 
beginning. 

As  you  develop  your  project  strategy, 
evaluation  should  be  a  key  component.  To 
design  an  evaluation  plan,  you  first  must 
identify  the  result  (outcomes)  you  want  to 


achieve.  You  must  determine  what  activities, 
services,  and  products  will  help  you  achieve 
the  desired  outcomes.  You  also  should  list 
the  resources  that  you  will  need  to  conduct 
the  activities  and  to  produce  the  products 
and  services  (outputs)  that  will  lead  to  your 
desired  outcomes. 

You  can  think  of  these  project  components — 
resources,  activities,  outputs,  and  outcomes — 
as  building  blocks  for  your  project  and  its 
evaluation.  One  tool  for  assembling  these 
building  blocks  is  known  as  a  "logic  model." 
A  logic  model  is  a  one-page  table  that  shows 
how  the  outcomes  you  want  your  program  to 
achieve  are  linked  to  the  resources,  activities, 
and  outputs  needed  to  achieve  them.  You  can 
also  think  of  logic  models  as  road  maps  that 
show  your  anticipated  destination  and  how 
you  expect  to  arrive  there.  With  this  map  to 
work  with,  you  can  use  evaluation  throughout 
the  project  to  help  you  stay  on  the  chosen  road 
and  make  course  corrections  as  needed.  Then, 
in  the  end,  you  will  be  able  to  assess  whether 
or  not  you  made  it  to  your  destination. 
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Step  One 


Develop  An  Outcomes-Based  Project  Plan 


The  logic  model  for  an  outreach  project 
should  be  developed  in  the  first  stages  of 
planning  your  project. 

Figure  1  shows  one  typical  logic  model 
template,  based  in  part  on  two  approaches: 
one  detailed  in  the  W.K.  Kellogg  Foundation 
Logic  Model  Development  Guide  [3]  and  the 
other  presented  at  the  University  of  Wisconsin 
Extension  Office  evaluation  Website  [4].  The 
W.K.  Kellogg  Foundation  guide  presents 
the  first  two  columns  as  the  part  of  the  logic 
model  related  to  project  plans  ("your  planned 
work")  while  the  last  three  columns  show  the 
outcomes  you  hope  to  achieve  ("your  intended 
results"). 

For  planned  work,  list  the  resources  you 
need,  including  personnel,  finances,  goods, 
technology,  or  anything  else  you  must 
obtain  for  the  project.  Be  realistic  about 
the  resources  you  will  need  to  conduct  your 
project.  The  activities  column  presents  the 
activities  you  plan  to  conduct  in  your  project. 

The  intended  results  are  then  identified  in  the 
last  three  columns.  First,  the  outputs  column 
describes  the  deliverables  of  your  project  or 


the  scope  of  your  project  (such  as  the  number 
of  people  trained  or  number  of  sessions 
conducted).  Second,  the  short-term  outcomes 
column  portrays  benefits  that  you  expect  your 
project  will  have,  often  describing  outcomes 
for  individual  participants  (e.g..,  increased 
knowledge  or  improved  satisfaction).  Finally, 
the  long  term  outcomes  column  is  used 
to  describe  community,  organizational  or 
system-level  changes.  Since  your  project  may 
only  last  12-18  months,  you  may  be  not  be 
able  to  achieve  system-level  changes  before 
the  project  ends.  However,  it  is  important 
to  articulate  long-term  outcomes  as  part 
of  your  overall  plan  so  you  understand  the 
"big  picture"  of  what  you  want  to  achieve. 
Completing  this  column  also  will  help  you 
decide  whether  you  need  to  collect  baseline 
data  about  changes  that  you  are  not  expecting 
until  far  into  the  future. 

The  Kellogg  Foundation  guide  also 
demonstrates  how  a  logic  model  actually 
presents  a  project  strategy.  In  Figure  1 ,  you 
see  a  series  of  if-then  statements  at  the  top 
of  each  column.  Together,  these  statements 
create  the  project  strategy  of  how  you  will 
reach  your  goals  [3]. 
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Table  1 :  Assumptions  in  Program  Planning 


Category 

Examples  of  Assumptions 

Target  population 

•  They  are  interested  in  your  activities 

•  They  can  be  motivated  to  participate 

•  They  will  be  available  to  participate 

Environment 

w       V^Uil \  C111CI11  allll  ICllctUlC  aCCCbo  LU 

computers  and  the  Internet  can  be 
obtained 

•    You  have  access  to  facilities  that 
are  suitable  for  your  activities  and 
convenient  to  the  target  population 

Staff 

•  Staff  members  expected  to  implement 
different  aspects  of  the  program  have 
the  skills  and  knowledge  to  do  so 

•  These  staff  have  the  time  and  resources 
to  fulfill  their  role  on  the  project 

•  These  staff  are  motivated  to  participate 
and  are  committed  to  the  project 

(your  outcomes)  before  deciding  how  to  get 
there.  Consider  the  short-term  and  long-term 
outcomes  early  in  the  planning  stage.  You 
might  run  a  logic  model  discussion  like  this 
(be  sure  to  write  with  something  erasable): 

1 .  Write  the  overall  goal  at  the  top  of  the 
logic  model.  The  goal  is  the  general 
purpose  of  the  project,  such  as  "To 
improve  community  members'  ability  to 
find  information  that  helps  them  manage 
their  health." 

2.  In  logic  models,  the  most  important 
columns  are  the  outcomes  columns  and, 
philosophically,  that  is  where  discussion 
should  start.  However,  most  people's 
first  ideas  are  triggered  by  activities  that 
they  intuitively  know  would  be  beneficial 
or  even  resources  they  need  for  those 
activities.  It  is  okay  to  start  with  the 
column  that  you  know  best  to  initiate 
conversation,  but  you  should  identify 
very  quickly  the  outcomes  you  expect 

to  accomplish.  Do  not  list  more  than  2 
activities  without  discussing  outcomes. 


The  University  of  Wisconsin  Extension 
Office's  logic  model  template  [4]  adds  two 
helpful  features  to  the  model  presented  in 
this  booklet.  As  you  work  through  the  logic 
of  your  project,  the  Assumptions  area  at 
the  bottom  of  the  logic  model  allows  you 
to  record  any  assumptions  you  have  about 
conducting  your  project  that  you  may  have 
to  confirm.  Table  1  shows  some  typical 
assumptions  that  outreach  teams  might  make 
when  planning  projects. 

The  Influences  area  should  contain  all  the 
positive  and  negative  influences  you  expect  to 
encounter  as  you  implement  your  project.  When 
possible,  assumptions  and  influences  should 
be  verified  through  pre-project  evaluation  and 
revised  as  needed. 

Developing  a  logic  model  is  a  little  bit  like 
brainstorming,  so  most  planners  do  not 
complete  the  logic  model  by  starting  in  the 
resources  column  and  moving  toward  the 
right.  In  fact,  if  you  think  of  a  logic  model 
as  a  map  to  a  destination,  you  can  see  that 
you  need  to  know  where  you  want  to  go 
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3.  Try  to  write  the  related  activities, 
outputs,  short-term  outcomes,  and  long- 
term  outcomes  at  the  same  time.  Add 
resources  as  you  think  of  them.  You 
might  want  to  use  column  breaks  to  show 
which  activities,  outputs,  and  outcomes 
belong  together. 

4.  After  you  have  completed  the  activities, 
outputs,  and  outcomes  columns,  think  about 
what  you  will  need  to  accomplish  your 
project.  This  information  belongs  in  the 
resources  column.  Column  breaks  may  or 
may  not  be  useful  here,  because  you  may  use 
the  same  resources  for  a  number  of  activities. 

5.  When  the  columns  are  completed,  identify 
the  factors  you  are  taking  for  granted  and 
list  them  in  the  assumptions  box. 

6.  Identify  the  factors  that  may  have  a 
positive  or  negative  effect  on  your  project 
(including  your  ability  to  obtain  funding). 
Are  there  resources  you  can  leverage?  Do 
you  have  data  that  documents  needs  of  the 
target  population  for  your  intervention? 
Are  there  any  barriers  you  may  be  facing? 
List  these  in  the  influences  box. 

Outcomes  and  outputs  are  sometimes  difficult 
to  distinguish.  Outputs  are  the  parts  of  the 
project  that  your  team  can  accomplish,  like 
the  number  of  training  sessions  you  plan 


to  conduct  or  the  number  of  people  you 
plan  to  train.  They  usually  are  the  project 
deliverables.  If  your  sentence  starts  with 
"we,"  such  as  "we  will  have  exhibits  at  10 
community  events,"  you  are  describing  an 
output. 

Outcomes  are  stated  with  an  emphasis  on  the 
project  recipients,  such  as  "the  participants 
will  increase  their  ability  to  find  information 
about  health  topics  they  hear  about  through 
the  media"  or  "  the  agency  staff  will  improve 
their  ability  to  find  health  information  for  their 
clients." 

Table  2  gives  some  categories  of  outcomes 
and  examples  that  can  be  entered  into  the 
outcome  columns  of  your  logic  model.  You 
will  notice  that  some  outcomes  are  related  to 
individuals  while  others  are  directed  toward 
change  at  an  environmental  or  community 
level.  If  your  anticipated  results  are  related 
to  individuals  and  you  expect  to  observe  the 
results  during  or  soon  after  your  outreach 
project,  they  probably  belong  in  the  short-term 
outcomes  column.  Place  results  targeted  at  a 
community  level  in  the  long-term  outcomes 
column.  You  also  might  put  anticipated  long- 
term  effects  for  individuals  in  this  column. 
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Table  2:  Examples  of  Different  Outreach  Outcomes 


Type  of  Outcomes 

Examples 

Individual  level 

Cognitive 

•  Increased  awareness  of  Internet-based  health  resources 

•  Imnroved  understandine-  of  side  effects  of  a  nrescrintion  drii£? 

A- A  lip  A.  v/  V  >v  \-4    UllUvl  ij  L  CI  1  lUUlCl    V/ A    JlUv    W  A  A  WW  IJ   V/ A    CI    Ul  VOV1  1L/  11V/1  1    Ui  v.* 

•  Improved  knowledge  of  how  to  control  a  chronic  health 
condition  like  hypertension  or  diabetes 

Affective 

•  Improved  confidence  in  finding  good  health  information 

•  Increased  confidence  in  asking  questions  of  a  physician 

Skills 

•  Improved  ability  to  distinguish  reliable  from  unreliable  health 
information 

•  Improved  ability  to  manage  health  issues  (e.g.,  prevent  asthma 
attacks;  cook  with  less  salt  to  manage  hypertension) 

Quality-of-care  outcomes 

•  Increased  use  of  Internet  resources  to  supplement  information 
provided  by  physicians  or  clinics 

•  Increased  use  of  health  information  when  making  health  care 
decisions 

Community  level 

Environmental 

•  Improved  community  access  to  the  Internet 

•  Improved  reliability  of  Internet  service  in  a  community 
organization 

Social 

•     Increased  number  of  volunteers  available  to  help  members  of 
the  community  access  online  health  resources. 

Ideally,  a  logic  model  is  created  with 
participation  of  representatives  of  all 
stakeholder  groups  (such  as  librarians, 
representatives  from  participating  agencies, 
and  clients  who  will  receive  services  through 
the  project).  In  reality,  the  first  logic  model 
may  be  drafted  by  a  team  submitting  a  grant 
or  contract  proposal,  then  revisited  later  by 
a  more  comprehensive  outreach  team  or 
advisory  group.  This  approach  is  fine,  as  long 
as  the  logic  model  is  viewed  as  a  flexible  plan 
that  can  be  revisited  by  a  larger  group  at  a 
later  time.  In  fact,  a  logic  model  should  be 


revisited  and  revised  periodically  throughout 
the  outreach  project.  Appendix  1  shows 
examples  of  how  logic  models  can  be  helpful 
at  many  stages  of  a  project. 

There  are  numerous  sources  for  designing 
logic  models,  with  many  available  on  the 
Internet,  including  materials  from  the  W. 
K.  Kellogg  Foundation  [3],  University 
of  Wisconsin-Extension  [4],  the  Institute 
of  Museum  and  Library  Services  [5],  the 
National  Network  of  Libraries  of  Medicine  [6] 
and  the  Free  Management  Library  [7]. 
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Step  Two 


Develop  an  Outcomes  Assessment  Plan 


Outcomes  assessment  allows  you  to 
demonstrate  the  results  of  your  project.  When 
you  make  a  claim  in  your  final  report,  you 
use  outcome  data  to  support  your  claim. 
For  example,  if  you  say  that  your  workshop 
increased  participants'  confidence  in  finding 
good  health  care  information,  you  would 
support  this  claim  by  writing  "The  percentage 
of  participants  indicating  they  were  'confident' 
or  'very  confident'  about  finding  good  health 
care  information  increased  from  45%  on  the 
pre-session  evaluation  to  80%  on  the  post- 
session  evaluation." 

The  first  step  in  writing  an  outcomes 
assessment  plan  is  to  identify  indicators  for 
each  short-term  and  long-term  outcome  in 
your  logic  model.  For  example,  a  short- 
term  outcome  of  a  training  class  might  be 

Table  3:  Outcomes  and  Indicators 


"participants  will  increase  their  ability  to  find 
out  about  their  medications."  In  this  example, 
ask  yourself  "What  outward  signs  can  be 
used  to  show  how  well  the  outcomes  were 
achieved?"  You  may  decide  to  have  a  quiz 
at  the  end  of  the  training  session  where  you 
give  participants  a  list  of  prescription  drugs 
and  ask  them  to  find  the  side  effects.  Their 
ability  to  answer  the  quiz  questions  correctly 
is  an  indicator  that  they  know  how  to  find 
information  about  their  own  medications. 
You  may  ask  them  directly  in  a  training 
session  evaluation  if  the  session  has  improved 
their  ability  to  get  information  about  their 
prescription  drugs.  The  number  of  affirmative 
responses  to  the  survey  question  is  an 
indicator.  Table  3  shows  some  other  examples 
of  outcomes  and  indicators. 


Outcome 

Indicator 

1 .    Participants  will  feel  more 
confident  about  locating  high 
quality  health  information  on  the 
Internet 

Participants  will  indicate  on  the  training  evaluation 
form  that  they  are  more  confident  about  locating 
high  quality  health  information  on  the  Internet 

2.    Diabetes  patients  will  discuss 
information  they  found  on 
MedlinePlus  with  their  diabetes 
educator 

Diabetes  educator  will  track  the  number  of  diabetes 
education  class  participants  who  bring  MedlinePlus 
information  to  discuss  in  class  or  at  appointments 

3.    Teenagers  will  use  MedlinePlus  to 
get  health  information  for  a  family 
member  after  they  receive  training 

Teenagers  will  indicate  in  a  survey  that  they  got 
MedlinePlus  information  for  family  members 

4.    Library  staff  will  use  NLM 

resources  more  often  after  being 
trained  on  these  resources 

Library  computers  will  show  more  hits  to  NLM 
resource  Websites  after  library  staff  has  been  trained 
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Once  you  have  identified  indicators,  you  now 
need  to  write  outcome  objectives.  Outcome 
objectives  set  a  target  and  timeframe  for 
success  of  your  program.  A  well-written 
objective  should  have  the  following: 

•  A  measurable  indicator  to  determine  if  the 
objective  was  accomplished, 

•  At  least  one  target  that  sets  your  criterion 
for  success,  and 

•  A  timeframe  for  when  the  objective  will 
be  accomplished 

Each  objective  should  be  achievable  given 
your  time  and  resources  and  the  priorities  of 
those  involved  in  the  project.  Table  4  shows 
outcomes  objectives  written  for  the  outcomes 
in  Table  3. 

Specifying  criteria  for  success  can  be  a 
challenge.  If  you  are  fortunate,  others  will 
have  conducted  projects  similar  to  yours 


Table  4:  Outcomes  and  Objectives 


Outcome 

Objective 

1.    Participants  will  feel  more  confident 
about  locating  high  quality  health 
information  on  the  Internet 

One  month  after  a  training  session,  50% 
of  participants  will  report  feeling  more 
confident  about  locating  high  quality 
health  information  on  the  Internet 

2.     Diabetes  patients  will  discuss 

information  they  found  on  MedlinePlus 
with  their  diabetes  educator 

Three  months  after  the  training  session, 
50%  of  diabetes  patients  trained  to 
use  MedlinePlus  will  report  having  a 
discussion  with  their  diabetes  educator 
about  the  information  they  found  on 
MedlinePlus 

3.     Teenagers  will  use  MedlinePlus  to  get 
health  information  for  a  family  member 
after  they  receive  training 

50%  of  teenagers  trained  to  use 
MedlinePlus  will  report  getting  health 
information  from  a  family  member 
within  a  month  after  training 

4.     More  NLM  resources  wil  be  used  after 
the  library  staff  have  been  trained  on 
these  resources 

There  will  be  a  25%  increase  in  the 
number  of  visits  to  NLM  resource 
Websites  from  the  library  computers 
six  months  after  all  library  staff  have 
completed  training 
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and  published  their  outcomes.  Their  work 
can  help  you  set  reasonable  expectations 
for  your  own  project.  A  second  approach 
is  to  get  feedback  from  stakeholders  as  to 
what  they  would  consider  an  acceptable 
"return  on  investment"  of  time  and  resources. 
Expectations  may  vary  based  on  how  long 
your  project  has  been  in  place.  If  you  are 
engaged  in  a  pilot  project,  small  changes  may 
be  adequate  because  your  primary  goal  is 
to  initiate  your  project.  If  you  are  involved 
in  expansion  of  a  pilot  project,  stronger 
results  may  be  expected  to  justify  continued 
investment  of  resources.  As  difficult  as  it  can 
be  to  specify  targets  for  your  results,  it  is  an 
important  exercise  in  setting  goals  for  your 
project  and  should  not  be  avoided.  However, 
be  careful  not  to  set  your  targets  too  high 
because  you  may  not  be  able  to  achieve  them 
given  the  typical  time  and  resource  constraints 
faced  by  outreach  teams. 
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Figure  2:  Evaluating  Findings  Using  Success  Criteria* 


Objective:  At  the  end  of  a  training  session,  50%  of  participants  will  report  feeling  more 
confident  about  locating  high  quality  health  information  on  the  Internet. 

Measurable  indicator:  %  of  participants  who  report  feeling  more  confident  about 

locating  high  quality  health  information  on  the  Internet 

Target:  50%  of  participants 

Timeframe:  One  month  after  the  training  session 

Data  Source 

Evaluation  Method 

Data  Collection  Timing 

Training  participants 

A  post-training  electronic 
survey  sent  to  all  training 
participants 

Participants  will  receive 
the  survey  approximately  1 
month  after  their  training 

*  Table  is  adapted  from  the  Institute  of  Museum 

Next,  put  together  a  more  specific  plan-of- 
action  for  evaluating  each  outcome.  You  will 
want  to  detailed  your  data  source,  evaluation 
method,  and  data  collection  timeline  to 
assess  your  indicator.  Data  sources  refer 
to  the  location  of  your  information.  Often, 
data  sources  are  people  (such  as  participants 
or  observers)  but  they  also  may  be  records, 
pictures,  or  meeting  notes,  (see  Appendix  2 
for  examples  of  data  sources).  Evaluation 
methods  are  the  tools  you  use  to  collect  data, 
such  as  a  survey,  observation,  or  quiz.  (See 
Appendix  3  for  examples  of  methods.)  You 
also  can  find  examples  of  sources  and  methods 
in  Measuring  the  Difference  (pages  60-63)  [1] 
and  in  Booklet  3  of  this  series. 

The  Institute  of  Museum  and  Library  Services 
provides  a  helpful  planning  tool  for  collecting 
indicator  data  [8].  Figure  2  is  adapted 
from  this  planning  tool  and  shows  how  the 
objectives  and  the  evaluation  plan  are  related. 

Data  only  make  sense  in  context,  so  you  also 
want  to  think  about  how  you  will  analyze  your 
findings.  There  are  two  basic  approaches. 
First,  you  can  compare  your  outcomes 
against  the  targets  you  listed  in  your  outcome 
objectives.  For  instance,  if  you  set  a  goal  that 


and  Library  Services  [8] 

80%  of  participants  in  your  outreach  project 
can  find  drug  information  by  the  end  of  your 
training  session,  then  you  can  analyze  your 
findings  (that  is,  assess  success)  by  comparing 
them  against  your  target. 

This  analytic  method  has  one  drawback.  If 
you  do  not  do  a  pre-project  assessment,  you 
do  not  know  how  many  people  could  find 
good  drug  information  before  your  training 
session.  Therefore,  you  cannot  show  how 
many  people  improved  because  of  your  project 
-  they  all  may  have  been  able  to  find  good 
drug  information  prior  to  training.  Your 
evidence  will  be  more  convincing  if  you  can 
show  change  over  time.  To  do  so,  you  must 
assess  the  level  of  indicator  before  you  start 
your  outreach  project.  This  is  known  as  a 
baseline  assessment.  Baseline  assessments 
are  helpful  anytime  they  can  be  conducted,  but 
they  are  essential  if  your  objectives  are  written 
like  Objective  4  in  Table  4  ("There  will  be  a 
25%  increase  in  the  number  of  visits  to  NLM 
resource  Websites  from  the  library  computers 
within  six  months  after  all  library  staff  have 
completed  training")  —  you  must  conduct 
a  baseline  assessment.  In  some  cases,  you 
can  still  get  baseline  data  after  your  project  is 
completed.  For  instance,  Web  traffic  reports 
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or  customer  databases  that  are  collected  on 
an  ongoing  basis  allow  you  to  compare  data 
before  and  after  the  program.  However, 
in  many  situations  you  will  lose  baseline 
information  as  soon  as  your  project  begins. 


So  be  sure  to  review  your  objectives  and 
evaluation  plan  to  see  what  baseline  data  you 
must  collect.  Figure  3  shows  an  example  of 
an  evaluation  plan  for  Objective  4  in  Table  4. 


Figure  3:  Evaluating  Findings  Using  Change  Over  Time 


Objective:  There  will  be  a  25%  increase  in  the  number  of  visits  to  NLM  resource 
Websites  from  the  library  computers  within  six  months  after  all  library  staff  have 
completed  training 

Measurable  indicator:  %  increase  in  the  number  of  visits  to  NLM  resource  Websites 
Target:  25%  increase 

Timeframe:  Six  months  after  library  staff  has  been  trained. 

Data  Source 

Evaluation  Method 

Data  Collection  Timing 

Web  traffic  data  from 
library  computers 

Pre/post  training 
comparison  of  number  of 
hits  three  months  prior  to 
training  and  number  of  hits 
three  months  after  staff 
training 

Total  number  of  visits  to 
NLM  resources  six  months 
prior  to  staff  training 
(baseline)  and  total  number 
of  visits  for  the  six  months 
after  staff  training 

*  Table  is  adapted  from  the  Institute  of  Museum  and  Library  Services  [8] 


Step  Three 


Develop  a  Pre-Project  Assessment  Plan 


The  outcome  evaluation  plan  is  designed  to 
evaluate  the  results  of  your  project.  However, 
to  plan  the  best  strategies  for  accomplishing 
your  outcomes,  you  will  need  a  pre-project 
assessment  plan. 

Pre-project  assessment  should  include  the 
following  steps: 

•  Revisit  any  community  assessment  data  you 
collected  before  you  developed  your  logic 
model; 

•  Investigate  the  assumptions  in  your  logic 
model  and  confirm  (or  discover  new) 
positive  or  negative  influences  that  you 
might  need  to  contend  with  in  your  project: 

•  Take  an  inventory  of  the  resources  you  have 
and  those  that  you  must  acquire  to  conduct 
your  outreach  project;  and 

•  Conduct  an  audience  analysis  for  the 
different  activities  in  your  logic  model. 


First,  look  at  the  community  assessment 
data  you  gathered  when  you  were  looking 
for  partners  and  communities  for  outreach 
projects.  You  may  already  have  information 
about  community  assets  and  resources  along 
with  the  general  readiness  of  the  community 
for  health  information  outreach.  After  you 
look  at  the  data  you  have  already  collected, 
determine  whether  you  need  other  information 
for  planning. 

The  assumptions  and  influences  usually  are 
assessed  by  interviewing  stakeholders  and 
project  participants.  You  also  may  review 
records,  such  as  attendance  at  events  held  at 
different  times  of  the  day  (to  see  when  activity 
is  greatest)  or  enrollment  numbers  in  different 
projects  offered  at  the  outreach  sites. 
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An  inventory  of  resources  may  involve 
interviews  and  site  visits.  It  also  may  include 
reading  the  literature  or  talking  with  other 
outreach  teams  to  get  an  idea  of  what  you  will 
need  for  your  project.  You  will  be  looking 
at  resources  like  staff  availability  during  the 
project,  meeting  places,  technology  centers, 
events  where  you  can  exhibit  online  resources, 
and  so  on. 

Audience  analysis  should  be  part  of  each 
structured  activity  (e.g.,  training  sessions)  you 
plan.  For  instance,  you  might  be  able  learn 
about  health  topics  of  particular  interest  to 
each  group,  so  you  can  tailor  your  activities 
to  include  those  topics.  You  will  want  to 
know  the  level  of  experience  and  comfort 
with  computers  and  the  Internet  for  the 


Step  Four 


Along  with  outcomes  and  pre-project 
assessment,  you  also  need  a  plan  to  monitor 
the  degree  to  which  you  implemented  your 
outreach  plan  as  well  as  the  quality  of  your 
activities  and  strategies.  In  other  words,  you 
will  be  monitoring  resources,  activities  and 
outputs  at  the  beginning,  during,  and  at  the 
end  of  your  project.  Keeping  track  of  your 
project  while  it  takes  place  is  usually  referred 
to  as  process  assessment.  It  has  two  purposes: 

1.  During  your  project,  process  assessment 
allows  you  to  find  out  early  about  any 
fixable  problems  or  challenges  in  your 
outreach  project  and  make  course 
corrections. 

2.  At  the  end  of  the  project,  you  will  review 
your  process  assessment  findings  to  help 
you  and  your  stakeholders  understand  why 
your  outreach  project  worked  or  did  not 
work. 


"average"  participant  in  your  training  session. 
Of  course,  you  will  always  have  people  at 
varying  levels  of  experience  and  ability  in  any 
group,  but  you  must  determine  and  design  for 
the  "average"  participant  if  you  are  providing 
some  form  of  structured  group  training.  (See 
the  information  about  stages  of  change  in  the 
first  booklet  or  in  Measuring  the  Difference, 
pages  28-29.  [1]) 

There  is  a  possibility  that  you  will  discover 
information  through  your  pre-project 
assessment  that  may  lead  to  modification  of 
your  logic  model.  After  you  have  completed 
this  pre-assessment,  you  may  find  it  necessary 
to  make  adjustments  in  either  your  activities 
or  your  expected  short-  and  long-term 
outcomes. 


To  assess  the  degree  of  implementation,  you 
will  want  a  plan  for  basic  record  keeping  such 
as  attendance  rates  at  training  sessions  or 
visitors  to  your  outreach  booth.  For  quality 
of  implementation,  you  can  collect  customer 
feedback  through  course  evaluations,  post- 
session  interviews,  or  focus  groups.  Table 
5  presents  some  typical  process  assessment 
questions  and  methods  for  addressing  them. 

Conducting  your  outreach  activities  will  be  a 
learning  experience.  Your  process  assessment 
may  lead  you  to  adjust  your  strategies  or  your 
expected  results.  Do  not  be  afraid  to  re-assess 
your  logic  model  and  revise  it  mid-way  through 
the  project.  However,  keep  track  of  earlier 
versions  of  the  logic  model;  seeing  how  it 
changes  can  be  part  of  the  process  assessment. 


Develop  a  Process  Assessment  Plan 
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Table  5:  Process  Assessment 


Question 

Methods 

How  closely  has  the  project  been 
implemented  according  to  the  plan? 

•  Number  of  training  sessions  completed 

•  Attendance  counts  at  training  sessions 

•  Number  of  visitors  to  an  exhibit 

•  Number  of  brochures  or  promotional 
items  distributed 

How  well  has  the  project  been 
implemented? 

•  Brief  post-session  evaluations 
completed  by  training  participants 

•  Interviews  with  trainers 

•  Interviews  with  activity  staff 

What  barriers  were  encountered? 

•  Interviews  with  trainers 

•  Interviews  with  managers  or  activity 
directors  of  agencies  that  host  training 

•  Interviews  with  participants 

Take  Home 
Messages 


Evaluation  must  be  incorporated  into  your 
overall  health  information  outreach  plan — 
before  the  project  begins,  while  it  is  going  on, 
and  at  the  end.  Your  logic  model  will  provide 
the  evaluation  plan  framework  for  the  pre- 
project,  process,  and  outcome  assessment.  If 
integrated  into  the  project  plan,  evaluation 
becomes  more  meaningful  and  useful  both 
to  the  outreach  team  and  stakeholders.  The 
steps  for  planning  a  project  with  an  integrated 
evaluation  plan  are  the  following: 

1 .  Develop  a  logic  model  with  participation 
from  a  diverse  group  of  stakeholders. 


2.  Write  measureable  objectives  based  on  the 
short-term  and  long-term  outcomes  columns 
from  the  logic  model. 

3.  Develop  a  reasonable  data  collection  and 
analysis  plan  for  outcomes  evaluation  based 
on  your  measureable  objectives. 

4.  Design  a  pre-project  assessment  to  assist 
with  project  planning. 

5.  Develop  a  process  assessment  plan 
to  monitor  the  degree  and  quality  of 
implementation  of  your  plan. 

6.  Think  of  the  logic  model  as  a  flexible 
document.  Revisit  and  revise  it  often  as 
your  project  develops,  but  keep  track  of 
earlier  versions  so  that  you  can  see  what  has 
changed. 
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How  to  Use  a  Logic  Model  at  Different  Stages  of  Your  Project 


Phase 

Benefits 

Who  should  be  involved 

Proposal  stage 

•  Allows  the  proposal  writers  to  plan 
thoroughly 

•  Helps  the  writers  anticipate  funders' 
questions  and  prepare  answers  for  inclusion 
in  the  proposal 

•  Provides  a  structure  for  writing  the  proposal 

•  Principal  investigator 

•  Project  implementers 

•  Evaluation  specialist 

Planning  Stage 

•  Brings  all  stakeholders  into  the  planning 
process 

•  Builds  relationships  between  outreach  and 
CBO  staff 

•  Allows  different  stakeholder  groups  to 
consider  assumptions  and  identify  potential 
barriers 

•  Helps  outreach  staff  find  ways  to  integrate 
into  the  CBO  services  and  projects 

•  Provides  a  benchmark  plan  against  which 
progress  can  be  evaluated 

Representatives  from  the 
following  stakeholder 
groups: 

•  All  partners  on  the 
outreach  team 

And  if  possible, 

•  Agency  staff  and 
volunteers 

•  Clients 

•  Evaluation 
consultant 

•  Advisory  committee, 
if  one  exists 

•  Funders 

Implementation 

stage 

•  Allows  group  to  check  actual  project 
progress  against  the  plan. 

•  Provides  convenient  method  for  updating 
new  team  members 

•  Allows  group  to  re-examine  assumptions 
and  influences  and  revise  plan 

•  Allows  alterations  in  short-  and  long-term 
outcomes  based  on  actual  implementation 

Note:  Good  timing  for  this  review  might  be 
quarterly  or  whenever  reports  are  due  to  a 
funding  agency. 

•  Outreach  team 

•  CBO  partners 
(anyone  involved  in 
project) 

•  Advisory  committee, 
if  one  exists 

Reporting  stage 

Provides  a  structure  for  reporting  evaluation 
data 

•  Compare  actual  implementation  against  plan 

•  Report  evidence  of  outcome  achievement 

•  Add  unexpected  outcomes 

•  Outreach  team 

•  Evaluation  consultant 

Note:  Feedback  about 
final  report  should  be 
obtained  from  CBO 
partners  and,  if  possible, 
participants 

Including  Evaluation  in  Outreach  Project  Planning 

Planning  and  Evaluating  Health  Information  Outreach  Projects.  Booklet  2 

Outreach  Evaluation  Resource  Center 
National  Network  of  Libraries  of  Medicine.  National  Library  of  Medicine.  2006 


15     Appendix  2 


Sample  Data  Sources 

People 

•  Participants 

•  Non-completers 

•  Project  implementers  (e.g.,  workshop  facilitators,  help  desk  staff) 

•  Administrators  or  agency  staff 

•  Volunteers 

•  Experts  (for  instance,  consultants  trained  to  review  training  materials  or  Websites) 

•  Advisory  committees 

•  Key  informants  (anyone  with  special  knowledge  about  a  project,  organization,  or  community) 

•  Community  leaders 

•  Funders 

Existing  Information 

•  Project  records,  such  as  attendance  sheets  from  workshop  sessions 

•  Meeting  minutes 

•  Written  material  kept  by  people  involved  in  a  program,  like  journals  or  notes  from  a  training 
session 

•  Email  or  bulletin  board  discussions 

•  Existing  databases,  like  demographic  information  about  participants  who  have  received 
services  from  an  agency 

•  Web  traffic  reports 

•  Newspaper  articles 

Observations 

•  Observations  of  users  navigating  Websites 

•  Videotapes  of  group  discussions  or  meetings 

•  Products  created  by  participants  or  stakeholders,  such  as  handouts  or  items  developed  by 
organizational  staff  to  promote  consumer  health  Websites 

•  Newsletters 

•  Pictures,  such  as  photos  taken  of  students  helping  their  parents  use  MedlinePlus 


Taylor-Powell  E,  Steele  S.  Collecting  evaluation  data:  an  overview  of  sources  and  methods. 

[Web  document].  Madison,  WI:  University  of  Wisconsin  Extension,  1996  [cited  26  June  2006] 
<http://cecommerce.uwex.edu/pdfs/G3658_4.PDF>. 
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Evaluation  Methods 

•  Surveys  -  questionnaires  administered  in  a  standardized  manner  to  collect  primarily 
quantitative  data. 

•  Interviews  -  question-and-answer  sessions  with  individuals.  Some  interviews  are  highly 
structured  while  others  may  be  very  unstructured. 

•  Focus  groups  -  interviews  with  a  group  of  people.  Usually  individuals  who  are  chosen  for  a 
group  share  some  similarity,  like  age,  profession,  level  of  computer  experience,  or  role  in  an 
organization  (managerial,  support  staff,  or  volunteer). 

•  Testimonials  -  these  are  brief  accounts  that  individuals  may  give  about  an  experience.  For 
instance,  you  may  ask  individuals  to  give  a  brief  account  of  how  they  used  MedlinePlus  the 
week  following  a  training  session. 

•  Logs  -  descriptions  of  events,  like  the  number  of  visitors  to  a  health  fair  exhibit,  or  participants 
at  a  training  session  or  a  daily  list  of  health  topics  brought  by  library  users  to  a  public  library 
reference  desk. 

•  Document  analysis  -  review  and  summary  of  written  material  pertinent  to  a  project. 
Examples  of  documents  includes  organizational  newsletters,  meeting  minutes,  or  existing 
evaluation  documents  created  during  an  organizational  self-study. 

•  Tests  -  exams  of  individuals'  skills  or  knowledge.  Tests  may  have  forced-choice  questions 
(multiple-choice  or  true-false)  or  open-ended  questions  such  as  "name  three  different  types  of 
information  you  can  find  at  MedlinePlus." 

•  Reflective  exercises  -  these  are  questions  that  are  designed  to  get  participants  to  reflect  on 
experiences.  They  may  be  used  on  a  one-time  basis  (like  at  the  end  of  a  training  session)  or 
they  may  be  used  to  structure  participants'  entries  to  a  journal  or  bulletin  board  discussion. 

•  Expert  or  Peer  Review  -  one  or  a  group  of  people  review  products  or  presentations  to  judge 
its  quality.  They  may  either  be  experts  (like  public  health  experts  reviewing  health  information 
promotional  materials)  or  they  may  be  peers  (like  outreach  librarians  observing  and  rating  their 
colleagues'  consumer  health  training  sessions).  Usually  the  review  process  is  structured  to 
assure  systematic  and  thorough  feedback. 


Taylor-Powell  E,  Steele  S.  Collecting  evaluation  data:  an  overview  of  sources  and  methods. 

[Web  document].  Madison,  WI:  University  of  Wisconsin  Extension,  1996  [cited  26  June  2006] 
<http://cecommerce.uwex.edu/pdfs/G3658_4.PDF>. 
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Senior  Center  Outreach  Project 

The  following  case  study  is  designed  to  demonstrate  how  to  develop  a  logic  model  and  an 
evaluation  plan.  For  this  case,  activity  directors  from  an  agency  that  provides  services  to  senior 
citizens  partner  with  health  science  librarians  to  design  a  health  information  outreach  project 
targeted  for  the  over-65  population  in  their  senior  program.  In  this  project,  MedlinePlus  training  is 
offered  to  seniors,  agency  staff,  and  family  members  of  the  seniors.  The  goal  of  the  program  is  to 
improve  seniors'  access  to  consumer  health  information,  both  by  improving  their  skills  and  training 
others  who  can  help  them. 

We  first  present  completed  worksheets,  based  on  the  case  example  described  here.  The  first 
worksheet  presents  a  logic  model  for  this  project.  The  second  worksheet  presents  three  objectives 
and  evaluation  plan  to  assess  outcomes  related  to  those  objectives.  The  third  worksheet  presents  a 
pre-project  assessment  plan  and  the  fourth  presents  a  process  assessment  plan.  Blank  versions  of 
these  worksheets  are  included  for  you  to  use  in  your  project  planning 
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Objectives  and  Methods  for  Assessing  Them 


Objective  1:  At  least  80%  of  seniors  who  receive  training  on  MedlinePlus  will  report  one 
or  more  things  they  learned  about  their  health  concerns  at  the  end  of  their  last  training 
session. 

Measurable  indicator:  Number  of  seniors  who  can  give  examples  of  what  they  learned 
Target:  80%  will  be  able  to  report  at  least  one  thing  they  have  learned 
Timeframe:  Immediately  after  the  last  training  session 

Data  Source 

Evaluation  Method 

Data  Collection  Timing 

Participating  seniors 

Individual  interviews  by 
facilitators  during  the  lunch 
session  that  follows  the 
class 

Immediately  after  last 
training  session 

Objective  2:  At  least  50%  of  trained  seniors  will  report  discussing  information  from 
MedlinePlus  with  their  health  care  providers  when  interviewed  three  months  after 
training 

Measurable  indicator:  %  of  trained  seniors  who  report  discussing  information  from 
MedlinePlus  with  their  health  care  providers 

Target:  50%  will  report  discussing  information  from  MedlinePlus  with  their  health  care 
providers 

Timeframe:  Three  months  after  training 

Data  Source 

Evaluation  Method 

Data  Collection  Timing 

Participating  seniors 

Survey  administered  by 
outreach  team  member 
(The  survey  will  be  read  to 
seniors  if  necessary) 

Three  months  after  last 
training  session 

Objective  3:  By  the  end  of  the  project,  at  least  25%  of  seniors  in  the  lunch  program  will 
have  one  family  member  who  has  attended  MedlinePlus  training. 

Measurable  indicator:  %  of  seniors  with  a  family  member  trained  on  MedlinePlus 
Target:  25%  of  seniors  will  have  a  trained  family  member 
Timeframe:  By  the  end  of  the  project 

Data  Source 

Evaluation  Method 

Data  Collection  Timing 

Training  attendance  records 

Family  members  will  be 
asked  to  identify  themselves 
on  training  session 
attendance  sheets 

Beginning  of  each  training 
session 

o 
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Pre-Project  Assessment  Plan 


Information  to  collect  for  each  area 

Plan  for  collecting  it 

Assumptions  and  Influences 

Deadline 

Confirm  the  following  assumptions  and  influences 

•  Seniors  and  staff  members  do  not  know  about  MedlinePlus 

•  Family  members  want  to  be  or  are  involved  with  the 
seniors'  health  care 

•  Most  participants  are  fluent  in  English 

•  Technology  labs  are  operable  and  Internet  connections  at  centers 
are  reliable 

•  Enrollment  of  daily  lunch  program  has  not  changed  since  the  last 
meeting  with  staff 

•  If  the  activity  director  has  changed,  check  the  willingness  of  new 
director  to  participate. 

•  Interview  staff  who  work 
closely  with  seniors  and 
family  members 

•  Interview  seniors 

•  Visit  facilities  to  examine 
computers 

•  Check  in  2-3  times  before 
project  begins  to  see  if 
Internet  connections  are 
working 

March  3 1 

Resource  Needs 

Deadline 

•  How  many  seniors  are  enrolled  in  the  lunch  program  at  each  center 
and  how  many  attend  regularly? 

•  How  many  family  members  are  likely  to  take  training? 

•  How  many  staff  members  will  be  trained  on  NLM  resources? 

•  How  much  time  can  outreach  librarians  spend  at  the  senior  centers? 

•  How  many  staff  members  from  the  centers  can  assist  in  training? 

•  What  access  do  activity  staff  members  have?  (Technology  center? 
Access  from  their  desks?) 

•  Do  seniors  and  their  family  members  have  Internet  connections 
outside  of  the  senior  center? 

•  When  are  the  best  times  and  days  for  scheduling  training? 

•  Request  attendance  data 
from  interviews  with 
administrative  staff 
members  who  keep  records 
of  attendance 

•  Interview  staff  who  work 
closely  with  seniors  and 
family  members 

•  Interview  staff  coordinators 
or  managers 

March  3 1 

Audience  Analysis 

Deadline 

•  How  interested  are  activity  staff  members  in  learning  to  use 
MedlinePlus  to  help  seniors? 

•  What  online  resources  do  the  activity  staff  members  use  when  they 
have  health  questions? 

•  What  health  topics  are  of  particular  concern  to  seniors? 

•  How  interested  are  seniors  in  learning  about  online  health 
information? 

•  How  likely  are  seniors  to  talk  with  activity  staff  about  their 
doctors'  visits? 

•  How  confident  are  they  in  talking  with  their  physicians? 

•  Who,  if  anyone,  in  their  family  helps  them  with  health  care  issues? 

•  How  likely  are  family  members  to  participate  in  training  sessions? 

•  Interviews  with  managers 
and  activity  staff  at  centers 

•  Interviews  with  seniors 

•  Interviews  with  family 
members 

April  10 
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Process  Assessment  Plan 


What  do  you  need  to 
monitor? 

How  will  you  monitor  it? 

When  will  data  be  collected? 

•  Participation  levels  at  each 

session 
Number  of  staff  trained 
Number  of  seniors  trained 
Number  of  seniors  with  at 
least  one  family  member  or 
loved  one  trained. 

Attendance  sheets  that  include 
name  of  participant  and  their 
relationship  to  the  center  (staff, 
senior,  family  member) 

At  each  training  event 

Quality  of  the  sessions 

Evaluation  sheet  given  at  the 
end  of  each  session 

Evaluation  sheets: 
End  of  each  session 

Interviews  with  two  seniors  and 
two  family  members  after  the 
training  session  for  feedback 

Interviews: 

Within  week  after  each  session 

Usefulness  of  the  information 
provided  in  training  session 

Mid-project  planning  and 
feedback  session  involving 
outreach  team,  trainers,  activity 
staff,  center  managers 

Interviews  with  two  seniors 
and  two  family  members  (see 
above) 

Six  weeks  into  program 
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Objectives  and  Methods  for  Assessing  Them 


Objective  : 

Measurable  indicator: 

Target: 

Timeframe: 

Data  Source 

Evaluation  Method 

Data  Collection  Timing 

Objective  : 

Measurable  indicator: 

Target: 

Timeframe: 

Data  Source 

Evaluation  Method 

Data  Collection  Timing 

Objective  : 

Measurable  indicator: 

Target: 

Timeframe: 

Data  Source 

Evaluation  Method 

Data  Collection  Timing 
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Pre-Project  Assessment  Plan 


Information  to  Collect  for  Each  Area 

Plan  for  Collecting  It 

Deadline 

Influences  and  Assumptions 

Inventory  of  Resources 

Audience  Analysis 
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Process  Assessment  Plan 


What  Do  You  Need  to 
Monitor? 

How  Will  You  Monitor  It? 

When  will  Data  Be  Collected? 

o 


Including  Evaluation  in  Outreach  Project  Planning 

Planning  and  Evaluating  Health  Information  Outreach  Projects.  Booklet  2 
Outreach  Evaluation  Resource  Center 

National  Network  of  Libraries  of  Medicine.  National  Library  of  Medicine.  2006 


Tool  Kit  —  Checklist  26 


Checklist  for  Booklet  Two  Including  Evaluation  in  Outreach  Project  Planning 


Step  One 

Develop  an  Outcomes-Based  Project  Plan 

□ 

Develop  a  logic  model  to  show  how  your  planned  work  (resources,  activities, 
and  outputs)  will  lead  to  your  intended  results  (short-term  and  long-term 
outcomes). 

□ 

List  your  assumptions  underlying  your  logic  model. 

□ 

Identify  the  positive  influences  that  will  support  your  project  and  the  negative 
influences  that  may  pose  barriers. 

Step  Two 

Develop  an  Outcomes  Assessment  Plan 

□ 

Identify  indicators  for  each  short-term  and  long-term  outcome  on  your  logic 
model  (indicators  are  observable  behaviors,  like  responses  to  a  survey  question 
or  performance  on  a  quiz). 

□ 

For  each  indicator,  write  an  objective  that  includes  a  target  (criterion  for  success) 
and  a  timeframe  for  achieving  that  target. 

□ 

Establish  where  you  will  collect  information  about  the  indicators  (e.g.,  among 
participants  or  from  documents  such  as  attendance  records),  how  you  will  collect 
it  (survey,  interview,  document  review)  and  when  (before  and  after  the  training 
session;  one  month  after  the  training  session). 

Step  Three 

Develop  a  Pre-Project  Assessment  Plan 

□ 

Look  at  your  logic  model  and  determine  what  pre-program  assessment 
information  you  must  collect  to 

-  Test  the  assumptions  and  confirm  the  influences  in  your  logic  model 

-  Assess  the  resources  identified  in  your  logic  model 

-  Understand  the  different  groups  you  will  be  addressing  in  your  outreach  efforts. 

□ 

Revisit  your  community  assessment  data  to  see  if  some  of  that  information 
answers  your  questions. 

□ 

Design  methods  for  collecting  the  pre-program  assessment  not  provided  in  the 
community  assessment  information. 

Step  Four 

Develop  a  Process  Assessment  Plan 

□ 

Trace  how  closely  your  project  is  being  implemented  according  to  the  plan  and 
how  well  you  have  achieved  your  deliverables  listed  in  the  output  column. 

□ 

Investigate  the  quality  of  your  project,  such  as  participant  training  session 
evaluation  forms  or  focus  groups  with  trainers. 
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